Selecting a Two-Group Classification Weighting Algorithm: Take Two

The two-group cross-validation classification accuracies of six algorithms (i.e., least squares, ridge regression, principal components, a common factor method, equal weighting, and logistic regression) were compared as a function of degree of validity concentration, group separation, and number of subjects. Therein, the findings of two previous studies were extended to the latter three methods with particular interest in how logistic regression faired as a function of validity concentration. In respect to validity concentration, as well as group separation and N, logistic regression was a mirror image of least squares. The same relative decrease, in respect to alternate methods, in accuracy with increasing validity concentration previously evidenced with least squares was observed. However, the large number of samples in which logistic regression failed to yield a solution may be a cause for concern. his investigation extends Morris and Huberty (1987) by contrasting the accuracy of six algorithms for classifying subjects into one of two groups. Ordinary Least Squares (OLS), Ridge Regression (Ridge), Principle Components (PC), Pruzek and Frederick’s (1978) common factor method (Pruzek), Equal Weighting (Equal), and Logistic Regression (LR) are the techniques that were compared. Darlington (1978) posited that regression cross-validation accuracy is a function of R 2 , N, and validity concentration, where R 2 represents the squared multiple correlation and N is the sample size. In Darlington’s formulation, validity concentration was used to describe a data condition in which the principal components of the predictors with large eigenvalues also have large correlations with the criterion. Thus, validity concentration requires at least a modicum of predictor variable collinearity (i.e., large predictor eigenvalues), but collinearity is only necessary, not sufficient, for validity concentration. Darlington suggested that the most useful statistical technique for practical prediction problems, therein, may be ridge regression. Through simulation, Morris (1982) re-examined the performance of Ridge with the same data structures on which Darlington posited the technique's superiority. With large validity concentration, Ridge was, indeed, more accurate than OLS, but contrary to Darlington's suggestions, Ridge was never the most accurate prediction technique. That is, in each case in which Ridge surpassed OLS due to large validity concentration, other contending weighting methods surpassed it. Morris and Huberty (1987) examined a subset of the methods (e.g., OLS, Ridge, and PC) considered in Darlington (1978) and Morris (1982) in the context of two-group classification accuracy, rather than regression, using the same simulated data conditions, but extending them to three different population model accuracies and two different sample sizes; not including the Pruzek nor Equal methods as in Morris (1982). The present study uses the same data conditions as in Morris and Huberty (1987), but includes the Pruzek and Equal methods. In addition, Logistic Regression (LR), an increasingly popular classification technique, was included. The same difficulties with multicollinearity (well-known with the OLS algorithm), have been mentioned (Hosmer & Lemeshow, 2000) as theoretically problematic to LR. Furthermore, ridge techniques have been suggested as a solution (Schaefer, 1986). However, the degree of effect of multicollinearity, and potentially resulting validity concentration, on the classification accuracy of LR has not been empirically and systematically examined.